Interactive agent assist mode

In interactive agent assist mode, unlike the passive and active modes, AI Agent is allowed to actively speak during the call. It plays audio to the call participants — the customer, the agent, or both — through named players, and may barge in on the ongoing conversation. This turns the "silent observer" into an active participant that can, for example, prompt the live agent or address the customer directly.

This mode can be used for the following use-cases:

Real-time coaching – whispering suggestions to the live agent that the customer does not hear.
Automated announcements – playing hold messages, disclosures, or confirmations to the customer.
Supervisor assistance – barging in to both parties when intervention is required.
Volume control – muting or attenuating one side of the call while a message is played to the other.

Configuring interactive agent assist mode

To configure interactive agent assist mode:

Start with configuring passive agent assist mode, as described in the passive mode section. Skip the Post call analysis configuration if it's not relevant for your use-case.
Navigate to the 'AI Agents > Agents' screen, locate the "agent-assist" agent that you created, and click Edit.
1. Switch to the Advanced tab and set the agent_assist.mode parameter to interactive:
```
{
    "agent_assist": {
        "mode": "interactive"
    }
}
```
2. Click Update.

Configuring players

A player is an audio channel to one or both call participants. Players are configured via the agent_assist.players parameter. Each player defines a list of targets (up to two), and each target specifies:

participant – customer or agent.
gain_db – (optional) playback gain, in dB, applied for this participant.
barge_in – (optional) whether the participant may barge in (interrupt playback).

The player's name is deduced from its targets: a player that targets only the customer is named customer, one that targets only the agent is named agent, and one that targets both is named all.

{
    "agent_assist": {
        "mode": "interactive",
        "players": [
            { "targets": [ { "participant": "customer", "gain_db": 0, "barge_in": true } ] },
            { "targets": [ { "participant": "customer", "gain_db": -10 }, { "participant": "agent", "gain_db": -10 } ] }
        ]
    }
}

If players is not configured, two default players are created: customer (targeting the customer) and agent (targeting the agent).

Playing to a player

For each configured player, AI Agent exposes a play_to_<player> tool to the LLM — for example, play_to_customer, play_to_agent and play_to_all. The LLM calls the tool to play a spoken message to that player. The tool has a single built-in parameter, text (the message to be spoken).

You can expose additional parameters to the tool via the agent_assist.play_tool_params parameter. Each entry defines:

name – the parameter name exposed to the LLM.
activity_param_name – the key the supplied value is written to in the played message.
type – str, int, bool or float (default str).
values – (optional) list of allowed values (enum); supported for str and int types.
description – (optional) parameter description shown to the LLM.
required – (optional) whether the LLM must provide it (default true).

For example, to let the LLM choose the TTS language and voice:

{
    "agent_assist": {
        "mode": "interactive",
        "play_tool_params": [
            { "name": "tts_language", "activity_param_name": "ttsLanguage", "type": "str", "values": ["de-DE", "en-US"], "description": "Language to speak in" },
            { "name": "tts_voice", "activity_param_name": "voiceName", "type": "str", "required": false }
        ]
    }
}

You can also attach static parameters to every message played to a given player via agent_assist.play_activity_params, keyed by player name:

{
    "agent_assist": {
        "mode": "interactive",
        "play_activity_params": {
            "customer": { "ttsLanguage": "de-DE" }
        }
    }
}

Handling plain responses

When the LLM produces a plain free-text response (that is, without calling a play_to_<player> tool), the agent_assist.default_player parameter decides what happens:

customer, agent or all – the response is played to that player.
none – (default) the response is discarded.

{
    "agent_assist": {
        "mode": "interactive",
        "default_player": "customer"
    }
}

Changing the volume between participants

Set the agent_assist.change_volume_tool parameter to true to expose a change_volume tool that adjusts the volume of the audio path between the two participants — for example, to mute one direction while a message is played:

{
    "agent_assist": {
        "mode": "interactive",
        "change_volume_tool": true
    }
}

The tool takes a direction (customer-to-agent or agent-to-customer) and a gain_db value (a signed integer, where 0 is normal volume and -32 is mute).

As in active mode, the agent_assist.no_response_phrases and agent_assist.trigger_words parameters can be used to reduce "chattiness" — see Active agent assist mode.

Testing an interactive agent

An interactive agent can also be reached by an ordinary call — a test call from the dashboard, or a phone number pointed at it. Such a call has a single participant and no players, so the agent answers it as a regular agent: it plays its welcome message and speaks to the caller.

The play_to_<player> tools stay available to the agent, and a message it plays with one of them is spoken to the caller — the player it named is left out. change_volume has no path between participants to act on, so it does nothing. Player and participant settings configured in session_params or activity_params are left out too — see Agent-assist-only call settings.

Use a test call to check what the agent says; check the routing between players on a real agent-assist session.